06. Demo: Defining Act Function and Experience Replay

Part 1

Cd13650 C5 L3 Demo 2 V3

Understanding the Agent Class Functions:

Explore the functionalities crucial to enhancing the capabilities of the agent class. This includes a look into:

1. Previous Setup Recap:

  • Design of the DQN architecture.
  • Initialization of loss and optimization functions.
  • Definition of agent parameters, including window size and feature count.

2. Act Function Concept:

  • Input: Receives a state representation at a particular time step.
  • Policy: Employs an ε-greedy approach to balance exploration and exploitation.
  • Output: Generates an action based on either a random selection or modeled prediction.

3. Random Action Selection:

  • Utilizes random.random to decide action based on ε.
  • Starts with 100% random action until ε decay adjusts probabilities.

4. Modeled Action Selection:

  • Computes Q-values using a helper function.
  • Uses numpy.argmax to choose the action with the highest Q-value.

Understanding and implementing these aspects are vital for enhancing AI's decision-making flexibility while minimizing overfitting.

Part 2

Cd13650 C5 L3 Demo 2b V3

Exploring the Experience Replay Function

Key Steps in Experience Replay:

  1. Inputs & Bellman Equation Implementation:

    • Input a batch consisting of the current state, action, next state, and reward.
    • Employ the Bellman equation for optimizing model fit towards the optimal Q-policy.

  2. Tracking Trading Loss:

    • Maintain an empty list to record trading losses, aiding in assessing model learning effectiveness.

  3. Mini Batch Preparation:

    • Initialize an empty list for the mini batch of memory.
    • Ensure the memory length does not exceed existing data.

  4. Memory Loop & Mini Batch Creation:

    • Extract the latest observations in training through indexing.
    • Form mini batches by iterating backwards through the memory.

  5. Defining Optimal Q Values:

    • Αpply the Bellman equation for optimization.

  6. Model Fitting with Target Q Table:

    • Generate Q-values for the current state, updating the Q table to serve as the target Q table.

  7. Track and Append Loss:

    • Leverage the fit function for model fitting and track training loss by appending it to the loss list.

  8. Implement ε Decay:

    • Conclude the process by establishing ε decay to adjust action strategies over time.

Part 3

Cd13650 C5 L3 Demo 2c V3

Reinforcement Learning Agent Overview

Purpose of ε Decay:

  • ε: Central to balancing exploration and exploitation during model training.
  • Decay Process: Applied post mini-batch training to encourage using learning feedback.
  • Exploitation vs. Exploration: Initial high epsilon values promote exploration; decay encourages the use of learned strategies.

Training and Model Functionality:

  • Model Training: Conducted using the exp_replay to deepen learning.
  • Parameters Set: ε and decay defined for adjusting exploration tactics.
  • Starting Values: Begin with ε = 1, decaying continually using epsilon
    decay rate.
  • Retention of Exploration: ε is never less than a predetermined minimum to maintain some exploratory behavior.

Agent Class Components:

  • Neural Network Design: Basic architecture created with Keras, allowing for expansion.
  • Function Definitions: Includes helper functions for reshaping vectors and managing built-ins.

Next Steps:

  • Model Training Demo: Upcoming demonstration focuses on training the complete reinforcement learning agent.

Next Concept